Using JABBERWOCKY to Achieve Corresponding Effects: Imitating in Context Across Multiple Platforms

We present a new approach to imitation learning of grasping skills in the context of a hu-man instructor who demonstrates a grasping action/skill to a mobile service robot equippedwith a 6-degrees-of-freedom arm, a dextrous hand, a multi-view active vision setup andmultimodal sensors. The proposed approach can be divided into 4 steps. The first step usesobject detection mechanisms to interpret the scene. Possible target objects are determined,as well as their position in 3D space.The second step performs data processing by observation of the skill demonstrated by theinstructor’s hand. Therefore the trajectory of the instructor’s hand, its orientation as well asits position, is acquired and filtered in 3D space. The service robot realizes this by a multi-view active vision setup using an adaptive contour tracker based on contour likelihoods. Atthe same time, to make the process more robust in complex situations, the instructor’s gazeis tracked to detect the object that he wants to grasp.In a third step, the acquired data from scene interpretation together with data from handand gaze tracking is then used to generate the trajectory of the robot’s arm and to se-lect a suitable hand configuration for grasping the object. The fourth step executes thegrasping action which is fine-controlled by visual servoing and force control. It is embeddedinto a reinforcement learning approach which evaluates the grasping action for robustnessand stability and represents the robot’s knowledge of which grasps are suitable for an object. K eywords: Imitation learning, grasping, service robot, multimodal interaction